The telecommunications world is relying more and more on Information Technology Systems to provide new functionalities. Thus, malware is a new threat to deal with. Malware families are increasingly adopting standard cryptographic suites like Secure Socket Layer (SSL) or Transport Layer Security (TLS) for their communication. Both, SSL and TLS protocols are based on the exchange of X.509 certificates.
Traditional network security tools such as IDS (Intrusion Detection Systems) or IPS (Intrusion Prevention Systems) have a good detection of abnormal and malicious behavior of the network elements when they work with non-encrypted traffic. However, detection systems suffer significantly in the presence of encrypted traffic, highly reducing their detection ratio. New trends show that some malware families have recently started to use standard encryption techniques (such as SSL/TLS) to avoid being detected by network security tools. Predictions say that this fact will be more and more frequent in the future.
Custom-made protocols are often used by malware to obfuscate their Command and Control (C&C) communications, but are typically easy to detect over the network because they are not used by benign traffic, which makes them easy to pick up (e.g. using network signatures). Also, writing secure cryptographic protocols is a hard task and malware authors often make mistakes that result in insecure (from their point of view) communications.
Having realized these limitations, developers of malware families are increasingly adopting standard cryptographic suites like SSL/TLS for their communications. The problem of malware using SSL/TLS is well known to the security community, however very little work has been done to really analyze this threat in a systematic way. Detecting this kind of malware is largely an open research question.
Malware families are adopting standard cryptographic suites like SSL/TLS for their communication. In particular, malware is increasingly using HTTPS (HTTP over SSL/TLS) protocol because it is a protocol widely used in benign web surfing and web applications. This, in turn, means that most protected networks will have their firewall open for outbound traffic on port 443/TCP, which is important for malware to be able to reach the Command and Control (C&C) infrastructure of said networks. Since HTTPS is widely used in benign network traffic it becomes challenging to distinguish malign HTTPS traffic from benign HTTPS traffic.
A current technique employed for improving the malware detection ratios with encrypted traffic network consists in adding high entropy detectors to an existing bot detection tool that uses deep packet inspection techniques for restoring bot visibility (Detecting Encrypted Botnet Traffic. Han Zhang, Christos Papadopoulos, Dan Massey, 2013). Document “No attack necessary: The surprising Dynamics of SSL Trust Relationships. Bernhard Amann et al. (2013)” discloses which degree benign changes to X.509 certificate ecosystems share structural properties with attacks. Document “Here's my cert, so trust me, maybe? Understanding TLS Errors on the Web. Devdatta Akhawe, et al. (2013)” discloses a study of the prevalence of different types of false warnings when browsers report TLS errors, providing a framework for said browsers to reevaluate their current warning mechanisms and conserve user attention.